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ABSTRACT 


Existing studies have shown that subjects tend toward the digitally rendered 
objects in an augmented reality (AR) environment. This tendency presents a potential 
problem for military use of AR systems where a user is faced with mission-critical 
decisions based on information presented through the AR headset. To reduce this bias 
toward digitally rendered objects, this thesis models four distinct experiments: reducing 
the fidelity gap, changing color, changing size, and altering the spatial layout, 
respectively. The goal is to identify bias of a human in an AR environment toward the 
rendered object. Furthermore, previous studies demonstrate that military members tend to 
look left first during search patterns. We seek to confirm this and provide additional 
quantitative data to confirm this conditioned behavior, taking into account that the 
American culture may predispose individuals to look left first, as in the case of a parent 


teaching children to look left, then right, before crossing a street. 
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CHAPTER 1: 
Introduction 





1.1 Introduction 

The Department of Defense (DoD) seeks to exploit new technology to enable warfighters 
to win on the battlefield. The technology under review in this thesis is augmented reality 
(AR). AR presents virtual objects to users which are novel when compared to the real 
world around the user. The purpose of this thesis is to layout a plan of research to provide 
groundwork for developers and military planners to properly implement AR into U.S. 
military formations. In this new research, we focus on the impacts of the fidelity, color, 
size, and placement of digital objects in an AR environment and an individual’s decision- 
making process. Specifically, we are interested in learning more about human choice and 
interaction with rendered versus real objects in an AR environment. Results from a previous 


thesis demonstrated participant bias favoring rendered objects over real ones. 


AR and virtual reality (VR) technologies reside on a continuum between the real and 
virtual environments; this continuum is called mixed reality (MR). These computing 
technologies and the explosive growth of the internet led to the creation of the field of 
Human Computer Interaction (HCI). HCI studies the design of computers, specifically the 
graphical user interface. Additionally, HCI studies how individuals use and are influenced 
by such systems [1]. Researchers continue to study unique ways individuals interact with 
computers from desktop computers, handheld devices, or wearables [2]. HCI has piqued 
the interest of the DoD [3] but little original research has been conducted on the impacts on 
the individuals using the equipment, the decision-making changes of the individual, or the 


psychological effects upon the individual. 


AR and VR offer an immense possibility to enhance learning through the use of visual 
perception and computer simulations. In 2017, Harvard Business Review outlined its 
reasons why every organization needs AR. Engineers, who traditionally compared complex 
paper blueprints in order to visualize the design of a ship or building, now can wear a pair 


of AR goggles and see a live three-dimensional (3-D) computer aided depiction of the same 


ship or building. In 2017, Harvard Business Review highlighted the revolutionary use of 
AR to enhance one’s understanding of internal components of a system, “AR applications 
provide a sort of X-ray vision, revealing internal features that would be difficult to see 
otherwise” [4]. In the medical field, the realized potential of AR to greatly enhance training 
of surgeons exists. AR has proven to be an effective technology allowing doctors and other 
medical staff to practice rare and complex procedures in a 3-D environment at low cost 
before a complex surgery [5]. Similarly, AR coupled with VR is used to rapidly prototype 
equipment, providing clients with 3-D visualization of yet to be built equipment. Previous 
and ongoing research for the U.S. Navy conducted at Naval Postgraduate School (NPS) 
visualizes a ships network with Microsoft’s Hololens 1, providing the ship’s captain with 
a visual display of the impact of a ship network outage due to maintenance or battle 
damage [6]. The U.S. Department of Homeland Security has started to implement AR and 
VR to conduct incident response training for emergency personnel [7]. The AR and VR 
market sector within Information Technology is rapidly growing; global analyst company 
CCS Insight predicts that the current 1.8-billion-dollar industry will balloon to 9.9 billion 
dollars by 2022, ushering in new uses, users, and applications for AR and VR platforms [8]. 


Advancements in digitally augmented and virtual environments are being adopted across 
civilian society as well as within the conventional ground forces of the DoD [9]. This tech- 
nology greatly expanded in the last decade as computer and graphics processors continue to 
be miniaturized and grow more powerful [10]. Simultaneously, businesses are instrumental 


in developing and adopting this technology. 


AR dates back to as early as 1968 with the work of Ivan Sutherland. Sutherland 
used a see-through head-mounted display (HMD) to display graphics to a viewer [11]. 
While working at MIT’s Lincoln Laboratory, Sutherland and his colleagues built an early 
Augmented Reality Head-Mounted Display (AR-HMD) apparatus called the “Sword of 
Damocles” (Figure 1.1) that consisted of an eyeglass display connected to the ceiling by the 
“sword” through which wiring to the main frame computer ran. The system was capable of 
head tracking, stereo perspective display, and a real-time see-through display that Sutherland 


used to display basic geometric shapes [11]. 





Figure 1.1. Ivan Sutherland's Sword of Damocles. Source: [12]. 


The aim of this thesis is to design four distinct experiments to examine the human 
selection bias in an AR environment. This thesis is based on previous research conducted 
at the NPS. Each experiment is designed to alter key variables (fidelity, color, size, and 


spatial placement) in order to understand the impact that each has on a user. 


1.2 Motivation 


There are numerous motivations for this thesis study. One was the author’s personal 
experience fielding hand-held devices to deploying forces, and performing | imited user 
evaluations with AR headsets designed for military applications such as route planning, 
navigation, and friendly force identification. Additionally, thesis research conducted at the 
Modeling Virtual Environments and Simulation Institute at the NPS by Maj. Donald Frisco 
in 2019 provide the start point for this follow-on research [13]. The author is motivated 
to understand the circumstances for human decisions in an AR environment and identify 
the experimentation necessary to determine if reducing the fidelity gap, changing the color, 
differing the size, or changing the spatial difference between the real and virtual objects 
impacts the user’s bias towards virtual objects. This is important to the author and to the 
DoD writ large to understand the impacts on human bias to ensure that future decisions are 
made without bias in order to protect the lives and missions of members of the DoD. 


NOTE: This thesis originally intended to study through experimentation the impact 
of fidelity in an AR environment had on human d ecision-making. H owever, due to the 
outbreak of the global pandemic, COVID-19, in the late stages of this thesis we lost the 
ability to conduct human experiments. As a result this thesis was refocused to develop a 
comprehensive plan for identifying influences on human decision-making in a Virtual En- 


3 


vironment (VE) to provide future researchers a plan for experimentation after the pandemic 
has subsided. 


1.3 Definition of Terms 
Definitions are adapted from Donald Frisco’s research in order to reproduce, repli- 


cate, and build on his research. Definitions have been refined while maintaining the core 
definitions as defined by Frisco [13]. 


1. Head-mounted display (HMD) -— In this thesis, this term refers to a head-mounted 
unit containing semi-opaque lens in the case of augmented reality systems and non- 
see-through screens that blocks out the real world in the case of virtual reality systems. 

2. Augmented reality (AR) — In this thesis, this term refers to the rendering of digital 
objects in real space that can be interacted with in real time via a remote, hand 
motions, etc. AR can be utilized via an AR-HMD or handheld devices. This research 
focuses on the AR-HMD systems. Examples of the handheld device for AR are 
IKEA’s AR mobile phone application to visualize new furniture in your home before 
buying [14]. Examples of other AR applications are the filters on popular photo 
applications within the Facebook and Snapchat apps. 

3. Virtual reality (VR) — In this thesis, this term refers to a fully immersive digital 
environment. Virtual digital objects can be interacted with in real time. VR compared 
to AR, replaces the real environment with a VE; additionally, the VR system may 
provide audio and other sensory feedback as well. 

4. Virtual object or Digital object — In this thesis, these terms refer to the computer 
rendered object that is displayed with an AR or VR system that can be manipulated 
and interacted with in real time. The object appears to the AR user as appearing in the 
real world through the display on the AR-HMD. The object also will appear through 


software on a computer mirroring the AR-HMD view. 


1.4 Research Questions 
This work pursues five research questions: 


¢ Does reducing the fidelity gap between a virtual object and a real object impact the 


human selection bias to physically engage one versus the other in an AR environment? 


4 


Does an object’s color affect a human’s selection bias if the real and virtual objects 


are otherwise indistinguishable? 


Does the object size affect human selection bias between a virtual object, rendered 
with a reduced fidelity gap, and a real object? 


Does the depth placement affect human selection bias of otherwise indistinguishable 
real and rendered objects? 


Can we validate previous demonstrations that military members visually attend to 
their left side first due to conditioned behavior? 


THIS PAGE INTENTIONALLY LEFT BLANK 





CHAPTER 2: 
Background and Literature Review 





Humans possess unconscious bias influenced by prior experiences and environmental 
affordances. The virtual environment presents new challenges in identifying the impacts 
of these biases and the positive and negative affordances provided by an AR environment. 
Through modeling, simulation, and experimentation, researchers have identified many 
biases inherent to particular demographics. Overlapping the virtual environment with this 
research, we seek to better understand a user’s situational awareness, cognition, and the 
impacts real-world and virtual biases have on the user. The concept of novelty in a virtual 
environment is well documented and has been shown to decrease interest through increased 
exposure [15]. Stimuli associated to novelty elicits exploratory behavior [16]. This research 
focuses on critical underlying attributes of the virtual object to understand their impacts 
on decision-making. Critical understanding of Wickens’s [17] and Boyd’s [18] decision 


models is necessary in order to frame the experimental models set out in this research. 


2.1 Human Bias 

In their work, published in Science magazine in 1974, Tversky and Kahneman present 
three heuristics, representativeness, availability, and anchoring and adjustment, that lead 
to biases [19]. Individuals rely on the representativeness heuristic when they evaluate the 
probability that A resembles B and vice versa. The frequency by which relative instances 
come to mind for an individual lead to a bias based on the availability heurisitic. Most 
specific to this research are “biases due to the retrievability of instances” [19] from memory 
and/or imagination. Additionally, illusory correlation is relevant to this research as the 
representation of one object influences the perception of the other. This correlation creates 
a bias affecting our judgement; the similarity between two items or events convinces the 
brain that the items are the same. Typically, illusory correlation refers to the erroneous 
perception of perceived relation between stimuli [20]. In the new environment of AR, 
information known prior to the user comes from the real world. AR environments simulate 
the real world and present familiar items virtually. Anchoring and adjustment heuristic, as 


Tversky and Kahneman describe, refers to the “people making estimates by starting from 


an initial value that is adjusted to yield the final answer’ [19]. AR users enter the new 


environment with prior knowledge based on the real world. 


Framing effects present a bias during experimentation. These framing effects bias the 
human decision process towards an answer or in the case of this research an object. Smith 
and Levin’s, in the Journal of Behavioral Decision-Making, [21], classified framing effects 
and the need for participant cognition when performing an action. Similarly, in Kahneman’s 
Thinking, Fast and Slow, he defines two systems for thought, system | and system 2. System 
1 is rapid decision-making based on instinct and emotion. System 2 is deliberate, logic 
based, and more time consuming method of thought. Solving simple addition problems, 
such as two plus two, or turning towards the direction of a loud sudden noise are examples 
of system 1. System 2 examples are more deliberate and calculated such as directing your 
attention towards an individual at a loud party or multiplying multiple numbers together. 
[22]. The more time a user has to think about their decision the “less contamination by 
biasing factors” [21] occurs. The same results were found when participants knew they 
would have to defend their actions [21]. These deliberate actions take additional time 
and overcome the inherent biases present in a situation through mental reasoning. Rapid 
decision-making does not afford this time. The more time a participant has to think about 


the decision the less effect biasing factors have [23]. 


Naturalistic decision-making is an approach focused on the theory that individuals use 
prior experience and previous situations to determine a decision [24]. Alexander Maule, 
in his research on computers aiding decision-making, defined naturalistic decision-making 
as“how decision makers overcome limitations in processing capacity based on the suggestion 
that, through expertise and experience, they learn to recognize situations and implement 
actions that are known to be effective in those situations’ [25]. In other words, when 
something is familiar to an individual it causes the individual to act in a certain way. 
This familiarity based upon experience influences the subconscious and is similar to the 
retrievability heuristic defined by Tverksy and Kahneman [19]. Another term for this is 
recognition-primed decision-making [24]. 


As the U.S. Military continues to implement new technology, users will gain in experi- 
ence that will shape future decisions. Understanding the underlying biases involved in these 


decisions is crucial — crucial in order to ensure that a timely, accurate, and safe decision is 


made in a high-stress military operation environment where human life, national security, 
and time is of the upmost importance. This brief overview of human psychology as it relates 
to bias provides a framework on which this new research is based. 


When modeling digital objects in an AR environment the user may have unconscious 
bias towards the digital objects. These biases are based on previous experiences [25] [24] 
and heuristics [19]. These potential biases were initially identified in Frisco’s research [13] 
as 87% of participants tended towards the virtual object. To determine if digital fidelity 
contributed to this bias, one experiment renders the digital objects in the most realistic 
manner given current technology. Additional experiments determine the impact on the 
bias that arises from color, size, and spatial placement of the objects. The results of these 
experiments provide the basis for recommendations to the DoD for the employment of such 
technology. 


2.2 Cyber Phenomenon 

The field of HCI has expanded since the early 1990s to include the psychological effects 
that computers have on humans. Mary Aiken, a leading forensic cyber-psychologist, is 
sought worldwide by law enforcement agencies to track down cyber criminals. Her work 
inspired the television series CSI: Cyber. Aiken leads the conversation on the impacts of 
the cyber world on humans. She believes that the technology is not the issue, rather, it is 
how the technology is implemented [26]. Through the visual cues, ubiquity, availability, 
and interaction techniques of computer systems, mobile, head-mounted, or otherwise, 
companies have developed interfaces that are addictive to their users, which promotes 
repeated and frequent use. This addictive situation presents a phenomenon that Aiken 
says is “hard to resist” given the “combination of fast delivery, exploring opportunities, 
[and] unexpected information...creates a medium that is enticing, exciting, and for some 
individuals totally irresistible” [26]. This addictive aspect does not mean that all technology 
will be the downfall of mankind, or in this case a Soldier, Sailor, Airman, or Marine on the 
battlefield. Military interface designers must design the technology in a way that mitigates 
addiction. Aiken discusses the impact the cyber world has on a human’s decision-making 
process. The cyber world and its interconnected nature to society has led to an online 
disinhibition effect (ODE), a term Aiken coins in her book, where individuals are both less 
inhibited and judgment impaired when interacting online. ODE will not be the focus of this 
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research, but it is important to understand that there are strong influences upon individuals 


within a virtual environment. 


The cyber and virtual environments are not synonymous. A virtual environment in this 
research’s context is the environment created through the use of AR and/or VR that facilitates 
human interaction with virtual objects and information. The cyber environment is overar- 
ching that includes the virtual environment as well as the internet, digital communication 


and information, and the digital interconnected nature of the modern age. 


One can infer that a virtual environment has an impact on the decision-making process of 
an individual. Designers of AR and VR systems for the military, understanding this impact, 
must exploit appropriate characteristics to provide the right information for decisions in 
the virtual environment while also mitigating the distracting or addictive characteristics. 
The interconnected virtual environment may create a new way of thinking, operating, and 
providing real-time information for service members on the battlefield. AR systems are 
currently used in military aircraft in the form of heads-up display (HUD) that provides 
bearing, speed, and other sensor data to the pilot and crew members. HUDs augment the 


pilot’s abilities through a wide network of sensors. 


New technology is presenting itself regularly and is in the hands of service members 
as evident by the signing of the $479,197,708.33 Integrated Visual Augmentation System 
(IVAS) contract awarded to Microsoft Corporation by the U.S. Army (USA). With this 
product, service members are discovering new and familiar means of interacting with the 
world that surrounds them and with other service members [9]. However, this contract 
comes before in-depth research has been conducted to understand the cyber-psychology 


impacting the service members on the battlefield. 


2.3 Digital Affordances and Color 


We perceive the environment through our senses and based on conscious and subcon- 
scious feedback. Humans make decisions on what they are able to do with the help of 
continuous sensory feedback. The concept of affordances was first presented by J.J. Gibson 
in the 1970s as a means to understand the subtle impact the environment, real or virtual, 
has on an individual. Gibson focused his study on the visual perception of the environment 


by animals and humans and what decisions the environment supports. As Dr. Gibson 
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defined them, “The affordances of the environment are what it offers the animal [human], 
what it provides or furnishes, either for good or ill.” [27] Objects, surfaces, positions or 
shape of objects, color of objects, and other physical properties in an environment relative 
to the individual provide feedback from which decisions are made. The perception of an 
object tells the individual something about it; a horizontal door handle, that is compress- 
ible, running the width of a door implies to a human that the door opens with a push. The 
shape and horizontal position of the handle affords a pushing motion to open. Positive 
affordances, such as the door handle, are seen throughout our environment and workplaces. 
Negative affordances are perceived by the individual as a threat, as un-useful, or are believed 
to be positive when they are initially encountered. Additionally, negative affordances are 
counter-intuitive; e.g., a push handle on a pull opening door. An example in design is the 
following image of stairs (Figure 2.1). When viewed, the individual recognizes that the 
stairs are there by the handles, but the steps are nearly indistinguishable due to the choice 


of carpeting that offers a danger warning feedback to the brain: 
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Figure 2.1. Negative Affordance Example. Source: [28]. 


12 


Training and repetition over time can change a human’s perception of the surrounding 
environment, but ingrained societal and evolutionary elements still exist. In U.S. culture, 
we learn from an early age that bright colors are used to grab our attention to things 
of importance; e.g., stop signs (red), firetrucks (neon yellow or bright red), highlighters 
(orange, yellow, pink). From our evolutionary biology, when we perceive threats we 
naturally execute what is described in literature as a fight or flight response [29]. Affordance 
theory is important in the field of human computer interaction as designers design virtual 
spaces for individuals to interact within. As humans alter the environment some affordances 
change. This is not a new environment, merely a modified environment [27]. The virtual 
environment is truly a new environment even if it is modeled from the real world that we 
live in. Designers of AR and VR environments must understand what affordances digital 
objects provide to the user. Affordances influence the decisions of an individual. If two 
cardboard boxes are present in the AR environment, one real and one virtual, it is critical 


to understand the impact the virtual object has on the user’s decision-making, if any. 


In previous research by Donald Frisco at NPS, he conducted an experiment to determine 
human preference between real and rendered objects. Using a low fidelity virtual cardboard 
box and a real cardboard box he measured the reactions of test participants. Specifically 
if they looked left first and which box they tended towards first. Sixty-five percent of 
participants looked left first. Eighty-seven percent of test subjects chose the virtual object 
first during a simple search pattern exercise [13]. An unknown element in this decision was 


the perception of the virtual object and why it was chosen first. 


Reaction time is impacted by color. Warm colors (reds and oranges) are received and 
processed by the brain faster than that of cool colors (blues, greens, and purples). This is 
because the wavelength these colors emit is received faster by the eye, optic nerve, and the 
brain [30] as well as the greater number of red cones than green or blue in the eye [31]. 
The millisecond difference in receiving provides the brain with increased processing time 
to react to the color. Through social training, such as stop lights, unconscious biases 
are formed in reaction to these colors. Red elicits an inhibition when compared to green 
changing a person’s mind when presented with color choices. Length of exposure to color 
stimuli before a decision is made can impact the effect bias has on the decision. Blizzard, 
Fierro-Rojas, and Fallah, in their research, postulate that the effect color has on higher level 


decision-making has more to do with the underlying human response inhibitions and not 
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bias [32]. Combining this research and the theories of affordance we seek to identify the 


influence color in an AR environment has on the user. 


2.4 Simulations and Situational Awareness 


MR environments provide a cost-effective tool for training in many fields. However, 
simulations are made of models and models are idealized representations of a phenomenon. 
Therefore, the simulated environment may represent, but not replicate, the real environment. 
A potential future use of AR technology includes providing information in real time to the 


service member to augment their world and aid in rapid real-time decision-making. 


In a limited study published in the MedEdPublish online journal, researchers looked 
specifically at the impact of high-fidelity imagery to provide fe edback to the us er. In 
addition to high-fidelity digital imagery, the researchers included olfactory and auditory 
components and measured the impacts of all three through post-experiment questionnaires 
using a Likert scale. This experiment returned mixed results, partly due to the small sample 
size (n=6). “Most markers suggested increased engagement within the augmented session. 
However, self-reported immersion was lower in this group when compared to the 
group taking part in the scenario without augmentation” [33]. Said another way, 
augmentation resulted in less perceived immersion. The researchers believe that high- 
fidelity simulation will boost subject involvement, knowledge retention, as well as 
reduce costs in the long term for the medical field. The recommendations for future 


research and the limited scope of the experiment in part motivated this thesis. 


Fatimah Lateef, a senior consultant for Singapore General Hospital, believes that simu- 
lation can enhance problem-solving and decision-making skills for practitioners in high risk 
environments; e.g., military and medicine [34]. The medical field has become a promising 
field of study as it relates to AR and V R. Medicine has invested heavily in simulation 
technologies for learning in an effort to boost knowledge retention and reduce the cost of 


iterative practice for medical practitioners. 


During his speech at the Association of the United States Army 2019 Annual Meeting, 
Secretary of the Army McCarthy outlined the vision of the Army for [VAS. The vision is to 
pair big data with artificial intelligence to increase the situational awareness of the soldier on 
the battlefield. By pairing these technologies, he sees “A system such as this, on target, has 
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the ability to rapidly identify a target, call in air support, de-conflict air space and neutralize 
a threat precisely in seconds.“ Additionally, the system provides a cost-effective method to 
conduct mission rehearsals in a simulated safe environment [35]. This integration presents 
a technological and possible psychological challenge for industry and the soldier. An 
AR system that boosts a user’s situational awareness could be lifesaving on the battlefield, 
allowing for better informed and faster decisions to be made. Alternatively, it could produce 
an unwanted situation where natural human biases are ignored and too much information 
is provided creating cognitive overload hindering a warfighter’s ability to act. Pushing the 
boundaries of technology to better enable the warfighter helps the United States Military to 
maintain its competitive edge but understanding how the service member’s decision-making 


is altered in this environment is crucial to understand before deploying the systems in mass. 


There are many perceived benefits to implementing AR and/or VR into a military 
setting. One such example is increased situational awareness as mentioned by the Secretary 
of the Army. Mega cities create a situation where the sheer numbers of noncombatants 
on the battlefield creates a unique challenge that adds tremendous stress on the warfighter. 
Mega cities reintroduce a widespread 3-D dynamic to the nature of warfare. Warfighters, 
once again, must look up (buildings) and down (subterranean), instead of focusing on the 
enemy to the front, left, or right of them. AR is seen as a likely candidate to assist the 
warfighter by providing near real-time visual information to increase situational awareness 
and understanding of the 3-D environment. “With the ability of AR to augment one’s view 
without obscuring that environment, AR became a natural paradigm in which to present 
military information. Head-up 3-D visualization within urban structures was considered 
a key benefit over two-dimensional (2-D) map visualizations” [36]. By communicating 
information to the user, via an AR-HMD from a leader or peer, the service member’s 
situational awareness is increased. Marine Corps Warfighting Publication 3-35 explains 
that communication increases situational awareness and has a positive effect on the mental 
state of the individual, especially in dark or confined environments. “Communication 


increases situational awareness and provides insight into the mental state of Marines” [37]. 


Communication in the medical field is well documented and researched. Communication 
increases overall situational awareness across the medical team and with the patient [38]. 
Shared perception and awareness of a situation leads to better and more deliberate deci- 


sions resulting in higher patient safety. [39]. In complex environments, such as medical 
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operating rooms and combat, no single individual of a team can process or know all of 
the pertinent information due to the distributed and complex nature of the environment. 
Technology helps reduce this complexity and improve situational awareness by enhancing 
distributed cognition between individuals. The technology brings the distributed informa- 
tion environment to a focal point by replacing verbal communication with technology aided 


communication [40]. 


There are many use cases of technology increasing situational awareness. Most notably 
is the recent use of tablets and smart phones on the battlefield in the counter-ISIS campaign, 
which provided U.S. and Coalition forces with a technological advantage over the ISIS 
fighters [41]. 


2.5 Information Overload and Distractions 

Counter arguments to using this digital augmentation for situation awareness and com- 
munication exist. The main one of concern proposes that it leads to information overload 
and distraction. The use of AR has the potential to mitigate some of this distraction by 
allowing the blending of reality with the virtual, but only if proper design is implemented 


and tactics are adapted to implement the system. 


There are three generally accepted types of cognitive load — intrinsic, extraneous, and 
germane. Intrinsic load is the inherent load that is the result of the complexity of the task. In 
this thesis we are interested in the extraneous which causes an individual to invest cognitive 
or mental resources to a task that are unnecessary or irrelevant [42]. The loads imposed 
upon a user of an AR system must not cause extraneous cognitive load if the system is used 
in a military situation requiring accurate timely decision-making. The final load, germane, 


is the result compounding activities required of an individual to complete a task. 


Military developers and planners must understand the impacts of AR on the service 
members’ decision-making and pay special attention to the design, layout, visual and/or 
auditory delay or latency, and overall usability in high stress environments. Current AR 
and VR systems are used by the civilian workforce in relatively lower stress environments as 
compared with the military. A system designed specifically for military use must consider 
the thresholds for information overload in order to not do more harm than good [36]. 
The US Army’s purchase of Microsoft Corporation’s Hololens 2 system [9] signals a 
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strong desire to apply current civilian AR systems to military applications. “The nature 
of the battlefield requires that the system be able to keep up with the chaotic nature of 
the battlefield, where unexpected events are common occurrences” [43]. To not distract 
the user or provide misinformation, traditional guidelines for design must be altered to 
adapt to the specific challenges of the ever-changing battlefield environment in order to 
provide accurate situational awareness augmentation. The Army Research Laboratory is 
researching visualization techniques specific to automated target recognition systems [44]. 
All of the inputs from the system will impact the user’s decisions, which may result in life or 
death. 


The aspect of distraction is often mentioned as a counter argument to implementing smart 
phones, AR, and other personally worn screen-based technology into the battlefield. John 
Spencer, the chair of the United States Military Academy’s Urban Warfare Studies, published 
an opinion piece based on observations of a tactical training simulation. The simulation 
evaluated the use of tablets and smart phones for real time critical information updates on the 
battlefield. In this experiment, the results were disastrous resulting in numbers of Soldiers 
being killed by a simulated enemy. Spencer proposes that introduction of technology on 
the battlefield has the potential to distract soldiers resulting in fatal consequences without 
the implementation of new policies and tactics to account for this. “I don’t want viral 
YouTube videos of distracted walkers bonking into light poles being replaced with videos 
of distracted soldiers walking into enemy bullets” [45]. Leaders must understand the impact 
that AR systems, human biases, and institutional training has on the user’s decision-making 
process when dealing with real and virtual objects. This research attempts to provide 
insights and a plan that leaders can use to better understand the conceivable positive and 


negative consequences of introducing AR into tactical military operations. 


2.6 Decision Models 


The interface and visual design of a system provide the medium through which an 
individual receives much of the benefits of technology. The study of HCI brings together 
computer science and cognitive systems engineering to team humans with machines through 
different interaction techniques and seeks to understand the optimal designs for a desired 
effect [1]. Cognitive systems engineering as it pertains to HCI is ecological and multi- 


dimensional and aims to provide the appropriate stimulus to the user in order to achieve 
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the task at hand [46]. Critical to this is deciding what counts as appropriate stimulus and 
what is unwanted stimulus that triggers a bias. Affordances and these stimuli are critical to 
ecological perception as described by Gibson [46]. HCI requires significant attention. In the 
realm of VR or AR, designers must be aware of potential distractions, misrepresentations of 
information, or providing negative affordances to the users impacting their decision-making 
process. As described in Section 2.3, simulations with either AR or VR can be used to aid 
individuals in training and rapid decision-making for a given situation. The foundational 
decision-making model common within the U.S. Military is Col John Boyd’s OODA loop 
(Figure 2.2) [18]. After his experiences in the Korean War, Col Boyd postulated his model 
of decision-making that has since been widely accepted within military and business circles. 
Four basic steps are the basis of the model: Observe, Orient, Decide, and Act, commonly 
referred to as the OODA loop. For example, an individual in a crosswalk observes a car 
approaching, the individual then orients towards the car either mentally or physically or 
both and calculates the options he has available to him to avoid the car. Then the individual 
decides upon one of the options and acts upon that action to execute the option. AR and 
VR environments visually present information to the user initiating the OODA loop cycle 
of the user. In order to gain an advantage over an adversary, one’s OODA loop speed must 
be faster, more accurate, and more efficient than that of the adversary [18]. In order to win, 
the military service member’s OODA loop must cycle faster (and more accurately) than his 
opponent in an effort to outpace the opponent’s OODA loop cycle and get inside of their 


OODA loop to disrupt or outpace it and win. 
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Figure 2.2. Col Boyd's OODA Loop. Source: [18]. 
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Service members operate in a world of conflict where reaction time is critical. If an initial 
observation and subsequent orientation take too long and hinder the overall speed of action, 
we posit that this will have negative, and potentially deadly, effects upon the warfighter. The 
goal of implementing AR into military environments is to enhance the warfighter’s ability 
to synthesize information and, therefore, speed up the warfighter’s OODA loop. To achieve 
this goal the AR system must display effective information to the warfighter. Information 
must be displayed in a format and visual representation so as not to promote underlying 
human bias. This represents a fundamental challenge. The challenge is developing the 
human computer environment that does not hinder the decision-making cycle. Currently, it 
is unknown if the current rendering of the digital objects does just that. Inferring from the 
results of Frisco’s previous research, we suspect that the differences between the real vs. 


rendered object biased the decisions of the individuals in some way [13]. 


The Von Neumann computer architecture was developed based on Von Neumann’s un- 
derstanding of how the human brain processes information. Similarly, Christopher Wickens 
modeled how the human brain processes information in his model of human information 
processing (Figure 2.3) explains through abstractions the flow of information processing by 


a human as envisioned by Wickens. 
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Figure 2.3. Christopher Wickens’s Human Information Process. Adapted 
from [17] 


Wickens outlined an interlocking sequence of events: Sensory processing, perception, 
cognition, memory, response selection and execution, continuous feedback loops, and 
attention. The model has no clear starting point because the human brain is continually going 
through this process; however, for the sake of discussion we will begin with perception. Data 
from the surrounding environment is relayed to the brain from the five senses and combined 
with information in the long-term memory of the brain. Through perception and cognition 
one can understand the situation and select a response for execution. Throughout the process 
there is continuous feedback to every step where re-evaluation can occur. Key elements 
to this process are the attention resources, as many mental processes are not automatic but 
require focused attention such as in the case of multi-tasking or when too many tasks are 
being performed resulting in divided attention [17]. This process is crucial in understanding 
the methods by which humans perceive their environment and subsequently make decisions. 


The U.S. Federal Aviation Administration outlines perception in the following manner: 
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We do not experience reality exactly as it exists, but as our experience and memories 
cause us to perceive it. Our sensory systems detect and take in stimuli from the 
environment in the form of physical energy. Each sensory receptor type is sensitive 
to only one form of energy. These receptors convert or transduce this energy into 
electrochemical energy that can be processed by the brain. However, our perception 
involves more than the receipt of sensory information. We must attend to, select, 
organize, and interpret this information in order to meaningfully recognize objects and 
events in our environment. [47] 


2.7 Summary 

Operating in the virtual environment has both risks and benefits. The interconnected 
nature provided by networked devices provides the rapid flow and synthesis of information; 
however, great care must be taken by designers to understand the human bias in decision- 
making especially in the virtual environment. The ease of use and availability of information 
may lead to dependency as laid out by Aiken. The rapid flow and availability of informa- 
tion can cause a dependency and/or a distraction to the users. This possible distraction or 
dependency is counter-acted by the opportunity for enhanced situational awareness as tech- 
nology facilitates the rapid availability of numerous data points. The increased situational 
awareness is balanced by the representation of data and ensuring that that data is devoid 
of decision influences that arise from the representation. Instead, the data should enhance 
the problem-solving and decision-making skills of the AR user. The visual representation 
of information is not devoid of affordances, both positive and negative. The background 
research on affordances is especially important to understand for this thesis to guide fu- 
ture application designers to avoid affordances that sway an individuals decisions whether 
that is fidelity, color, size, or position. In this research, we seek to understand the human 


decision-making in a selection process. 


Two decision models are presented to provide the reader with a framework to understand 
how individuals make decisions regardless of their environment. An understanding of these 
models is important, especially to AR application developers, to prevent AR users from 
becoming “stuck” at any one phase of the decision-making process. Unnecessarily halting 
at a phase can have lethal effects in a military environment. 


In conducting the research for this thesis and the underlying experiments we found no 


existing research studying the impact on an individual’s decision-making process when 
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conducting operations in a MR environment either with AR or VR with the exception of 
Frisco’s research of which this research is a follow on [13]. This impact must be understood 
to provide interface designers with guidelines on design and to equip military leaders and 
acquisitions experts with knowledge of the impacts this technology will have on military 


service members in dangerous and often deadly military situations. 


pie 





CHAPTER 3: 
Methodology 





In order to maintain the integrity of the experiments and provide validation of previous 
work, the following experiments seek to replicate and follow on from the experiment in 
Frisco’s thesis “Real or Rendered? Determining Human Preferences in an Extended Reality 


Environment” [13]. 


Four distinct experiments are developed here to measure the human selection bias in an 
AR environment. The experiments are designed similar to Frisco’s with each experiment 
changing one variable at a time to reduce possible confounds. The key variables are fidelity, 
color, size, and spatial positioning. All of the experiments in this thesis utilize a real and 
a virtual object, namely a real and virtual cardboard box. All experiments are designed 
recognizing they will be conducted within the constraints of the AR platforms and the 


physical room layout. 


In experiment one, the fidelity experiment, a virtual cardboard box is rendered in a 
manner that is nearly indistinguishable from the real cardboard box. We recognize that 
“nearly indistinguishable” is reliant upon the constraints of the AR platform, the ability of 
the author to configure the virtual object in Unity, and subjective. The indistinguishability 
is a topic for future research and experimentation. In rendering the boxes in this manner we 
seek to reduce or remove the novelty presented by the virtual box. The hypothesis is that 
the human selection bias towards the rendered object is reduced if the rendered object is 
less distinguishable from the real object If a decrease is not observed, it is inferred that the 
fidelity difference between Frisco’s research and this research is not enough to reduce the 
bias. If the human selection bias is towards the virtual object, we expect that this bias will 
carry forward to the follow-on experiments. This compounding bias is a topic for future 


research. 


Experiment two is the color experiment. In this experiment the color variable that is 
explored in order to identify the effect changes in color has on the human bias, in an AR 
environment, towards objects that are rendered otherwise indistinguishable with the current 


AR software and hardware. The texture and design are carried forward from experiment 
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one to experiment two. The hypothesis for this experiment is that color induces a human 
selection bias towards the colored object (>50% of the time), regardless if the object is 
real or virtual. The outcome of the color experiment has no impact on the following size 
experiment, but the results will be used in future research combining multiple variables. 
Experiment two is composed of 25 iterations. Through these iterations, numerous data 
points are collected and familiarity is gained by the user potentially reducing the impact 


novelty the virtual object has on the user. 


Experiment three, the size experiment, closely follows the same parameters of the fidelity 
experiment as it pertains to rendering of the virtual object with size being the variable 
manipulated. The hypothesis for this experiment is that the participant selects the larger of 
the two objects (>50% of the time), regardless if the object is real or virtual. The outcome 
of the size experiment has no immediate impact on experiment four; however, the results 


will be combined in later experiments. 


Experiment four, the depth of placement experiment, follows the same parameters of the 
fidelity experiment as it pertains to the rendering of the virtual object with the placement 
of the boxes changed within the room. The farther object is placed twice as far away as the 
near object. The hypothesis for this experiment is that the participant will select the closer 


of the two objects (>50% of the time), regardless if the object is real or virtual. 


The Bernoulli Trial was used to initialize the hypothesis >50% of participants would 
choose one object over the other in each experiment. Each trial, or experiment, has an 
outcome between two objects: real or virtual, colored or not colored, larger or smaller, 
closer or farther respectively. If we let X equal | if the outcome of each experiment is the 
first of the two outcomes and 0 if the second, then the probability mass function of X is 


given by: 


p(O) = P{X =O} =1-p 


p(1)=P{X=I1} =p 


where p, 0 <p< 1 , is the probability that the trial is the first outcome. X is the Bernoulli 


random variable if its probability is defined as the above function for some p € (0, 1) [48]. 
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Each experiment will continuously look to validate the previous demonstrations that 
military members visually attend to the left side first due to conditioned behavior. The 
following research question is asked during each of the four experiments: Can we validate 
previous demonstrations that military members visually attend to their left side first due to 


conditioned behavior? 


The author’s hypothesis is: Yes, military members will look left first. The focus of this 
study is with the military community but recognizes that the U.S. Military may be reinforcing 
the American culture of looking left first such as when crossing a road. Additionally, U.S. 
Military training reinforces this automated behavior. 


With each experiment the concept of novelty arises as a possible confound. The virtual 
box is less familiar to each participant than a real box. It is assumed that the participants are 
familiar with and have interacted with real boxes on numerous occasions throughout their 
life. As familiarity is increased the novelty is reduced [15]. Experiment two, through 25 
iterations, has the potential to identify if novelty impacts the participants decision between 


selecting the real or virtual box. 


Figure 3.1 represents the information flow between experiments and goals for each. 
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Figure 3.1. Experiment Model 


25 


3.1 Experiment Design 

The experiments one through four were planned to be conducted with a group of >65 
test participants. Within a 30-minute session, the participants will complete a Consent and 
Direction Form and Pre-Test Questionnaire, they will receive directions on how to wear the 
AR-HMD and basic interaction techniques with the device. Following that, participants 
will execute the experiment and complete Post-Test Questionnaires. The AR-HMD being 


used in this line of research is the MagicLeap | (Figure 3.2). 


NOTE: With the delivery of the Hololens 2, we expect to broaden the spectrum of future 
research to include different AR devices if they provide a capability that directly addresses 


any of the variables of interest in this experimental plan. 





Figure 3.2. Magic Leap One 


The Consent and Pre-Test Forms do not reveal the data to be collected in order to prevent a 
bias from being developed by the participant. They outline the task to be performed, namely 
conduct a search in a room with an AR-HMD for a real or digital rubber duck hidden in a 
box in the room. The rubber duck will not actually exist, but this prompts the participants 
to explore the room and allows us to capture which box they visually and physically engage 
using the two measures of gaze and directed motion. A 5-minute familiarization with the 


AR-HMD system is allotted to the subject prior to the experiment execution. During this 
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familiarization, the participant is shown how to interact with virtual objects. During this 


time the participant is afforded the chance to ask questions. 


The participant is given two minutes to complete the experiment task. The participants 


enter the room, which has one real and one virtual box inside. 
The universal general variables for each experiment are as follows: 


1. Variable: Scan. Do the participants scan the room? Yes/No. 

2. Variable: Gaze. Which object do they visually assess first? Left/Right. 

3. Variable: Time. The time it takes them to move toward an object. Time is tracked 
consistently in the following manner. Time-tracking begins when participants are told 
to enter the room and time stops when the participants move to within three feet from 
an object. A three-foot marker is placed around the objects. Once the participant 
moves within the three-foot area, the test is complete. 

4. Variable: Directed Motion. Which object do they move towards first? Real/Virtual. 


Both the scan and gaze variables are binary due to the simplified nature of the environ- 
ment. The experimentation room only has the real and virtual boxes placed inside. A single 
scan of the room allows the experiment participant to see both items in the room and view 


the entirety of the room. 


The unique parameters of each experiment are explained in Sections 3.1.1-3.1.4: 


3.1.1 Experiment One: Fidelity Difference Experiment 
Experiment one follows directly from Frisco’s “Real vs. Rendered” experiment as seen 


in Figure 3.1. 


This experiment (Figure 3.3) asks the research question: Does reducing the fidelity gap 
between a virtual object and a real object impact the human tendency to physically engage 
one versus the other in an AR environment? Additionally, what percentage of participants 


visually attend left or right? 


Hypothesis: Human bias towards engaging with rendered objects is reduced if the 
rendered objects are less distinguishable from the real object. 


Zi 


The expected behavior for the experiment one is that when the participant enters the 


room, > 65 % of participants look left, and < 87 % moving towards the virtual object. 


This hypothesis is undermined if the number of participants choosing the virtual object 
remains the same or increases. Therefore, the digital rendering has less, or minimal, effect 


on the decision of an individual towards the virtual object. 
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Figure 3.3. Experiment 1: Fidelity Difference 


One virtual and one cardboard box are placed in the room. The virtual box is rendered 
to a higher fidelity than Frisco’s previous experiment [13] in order to determine if the 
decreased fidelity difference between the real and rendered objects impacts the human 
decision-making towards the virtual object. Frisco discussed that participants tended to 


engage a virtual object first 87% of the time. 


Both boxes are placed outside of the participant’s field of view (FOV) when the user first 
enters the room, see Figure 3.4. The boxes are placed far enough apart so the participant 
must move their head in order to scan the room; this allows the test proctor to observe 
and annotate whether the participant looks left first. The test proctor is situated behind 
the participant near the doorway allowing for full view of the participant throughout the 
experiment. The position of the real and the virtual boxes are randomly switched prior 
to each experiment. Randomization is done to account for a participant speaking their 
reactions and waiting participants overhearing in order to ensure accurate measurements of 


future participants’ actions. 


For each experiment, specific data is recorded during each iteration to capture the specific 


room and environment set up. For experiment one, this includes the following data: 


1. Do the participants scan the room? Yes/No. 
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2. Which object do they visually assess first? Left/Right. 
3. Time it takes them to move toward an object. 
4. Is the virtual Box on right or left? 


The data collected on each iteration of this experiment provides data for statistical 
analysis and highlights information for future experiments where multiple variables are 
changed. This experiment precedes experiments two, three, and four, providing baseline 
understanding of the impact of fidelity on human decision-making. 


Fidelity Experiment Room Layout Example 


Subject’s FOV 


Outside of FOV Outside of FOV 


Subject 





Figure 3.4. Fidelity Difference Experiment Room Layout 


3.1.2 Experiment Two: Color Difference Experiment 
Experiment two follows directly from experiment one. 


This second experiment (Figure 3.5) asks the research question: Does an object’s color 


affect a human’s selection bias if the real and virtual objects are otherwise indistinguishable? 


Hypothesis: Bias is towards the colored object (>50 %) regardless of the type of object. 
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The expected behavior is that when the participant enters the room, > 65% of participants 
look left and >50 % of the participants engage the colored object first regardless whether it 


is real or virtual. 
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Figure 3.5. Experiment 2: Color Difference 


The room is prepared in the same manner as the experiment one with one virtual and 
one real cardboard box. Color options are as follows, cardboard material, dark red (RGB: 
128, 0, 0), bright red (RBG: 255, 0, 0), dark green (RGB: 0, 128, 0), or bright green (RGB: 
0, 255, 0) between each execution of the experiment. The boxes are laid out as shown in 
Figure 3.6. Boxes of different colors are created virtually utilizing Unity for the Magic 
Leap |. Five real cardboard boxes are required for this experiment. One of each of the 
following: original brown cardboard, dark red, bright red, dark green, and bright green. 
Each participant executes the experiment 25 times (Table 3.1). In this experiment, the visual 
assessment (left vs. right) is only measured during each execution of the experiment for 


each participant 
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Table 3.1. Experiment Two Iterations 




























































































Iteration # | Real Box Color | Virtual Box Color 
1 Cardboard Cardboard 
2 Bright Green Cardboard 
3 Dark Green Cardboard 
4 Bright Red Cardboard 
5 Dark Red Cardboard 
6 Cardboard Bright Green 
f | Bright Green Bright Green 
8 Dark Green Bright Green 
9 Bright Red Bright Green 
10 Dark Red Bright Green 
11 Cardboard Dark Green 
12 Bright Green Dark Green 
13 Dark Green Dark Green 
14 Dark Red Dark Green 
15 Dark Red Dark Green 
16 Cardboard Bright Red 
17 Bright Green Bright Red 
18 Dark Green Bright Red 
19 Bright Red Bright Red 
20 Dark Red Bright Red 
Zi Cardboard Dark Red 
22 Bright Green Dark Red 
23 Dark Green Dark Red 
24 Bright Red Dark Red 
25 Dark Red Dark Red 








This experiment collects data on each participant when they are presented with the same 
item i.e., two cardboard boxes of the same color, to add additional data for experiment one. 
Additionally, Results from these four iterations may provide information that leads to an 


additional experiment. 
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For each iteration of the experiment two, specific data is recorded to capture the room 


and environment set up. For experiment two, this includes the following data: 


. Do the participants scan the room? Yes/No. 

. Which object do they visually assess first? Left/Right. 
. Time it takes them to move toward an object. 

. Virtual Box on right or left? 


nA FW NHN 


. What iteration is being executed. 


The data collected on each iteration of this experiment provides data for statistical analysis 
and highlight information for future experiments. This experiment is independent of the 
following experiments; however, data collected here is pertinent to future combinations of 
variables in the future experiments. Possible future work includes arranging multiple boxes 
of different colors in front of the participant and identifying which colors the participant 


tends towards the most. 


Color Difference Experiment Room Layout Example 


Subject’s FOV 


Outside of FOV Outside of FOV 


Subject 





Figure 3.6. Color Difference Experiment Room Layout 
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3.1.3 Experiment Three: Size Difference Experiment 
Experiment three follows directly from experiment one and can be executed without the 


information from experiment two. 


This third experiment (Figure 3.7) asks the research question: Does the object size affect 


human bias between a virtual object, rendered with a reduced fidelity gap, and a real object? 


Hypothesis: The size of the object does not affect the human selection bias towards 
rendered objects; the human bias is towards larger objects (>50 %) regardless of the type 


of object. 


The expected behavior is that when the participant enters the room, > 65 % of participants 
look left and >50 % of participants engage the larger object first regardless of whether it 
is real or virtual. This engagement may be based on learned early infant development 
behaviors [49]. 







2% Look Left 
2% Look Right 


Size Difference 


Bias % 


Large Small 


Figure 3.7. Experiment 3: Size Difference 
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The room is prepared in the same manner as the experiment one with one virtual and one 
real cardboard box. Each iteration of the experiment is outlined in Table 3.2. Additionally, 
data is collected when the participant is presented with two of the same boxes (Tables 3.3and 
3.4). The boxes inside of the room will differ in size by a scale of 1:2 (Figure 3.8). One large 
and one small real box and one large and one small virtual box are used for this experiment. 
The size selection is randomly selected between each experiment. Each participant executes 
the experiment four times. In this experiment, the visual assessment (left vs. right) is only 


measured during each execution of the experiment for each participant. 


Table 3.2. Experiment Three Iterations 





























Iteration # | Real Box Size | Virtual Box Size 
1 Small Big 
2 Small Small 
3 Big Small 
4 Big Big 





Table 3.3. Experiment Three Iterations for the Real Box 




















Iteration # | Real Box 1 Size | Real Box 2 Size 
1 Small Big 
2 Big Small 








Table 3.4. Experiment Three Iterations for the Virtual Box 




















Iteration # | Virtual Box 1 Size | Virtual Box 2 Size 
1 Small Big 
p) Big Small 








For each iteration of the experiment three, specific data are recorded to capture the room 


and environment set up. For experiment three, this includes the following data: 


1. Do the participants scan the room? Yes/No. 
2. Which object do they visually assess first? Left/Right. 
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3. Time it takes them to move toward an object. 
4. Virtual Box on right or left? 
5. Size of virtual box: Large or Small. 


6. Size of real box: Large or Small. 


The data collected on each iteration of this experiment provides data for statistical 
analysis and highlighting information for future experiments. Collecting data using two of 
the same box in the room is to collect data in order to understand further underlying biases 


affecting human selection. 


Size Difference Experiment Room Layout Example 


Subject’s FOV 


Outside of FOV Outside of FOV a 


Figure 3.8. Size Difference Experiment Room Layout 











3.1.4 Experiment Four: Spatial Difference Experiment 


Experiment four follows directly from experiment one. 


This fourth experiment (Figure 3.9) asks the research question: Does the depth placement 


affect human selection bias of otherwise indistinguishable real and rendered objects? 


Hypothesis: Humans will tend to engage with closer objects first (>50 %). 


a5 


The expected behavior is that when the participant enters the room, > 65 % of participants 


look left and >50 % engage the closer or the two objects first. 


2%— > Look Left 
2%) Look Right 






Spatial Difference 


Bias % 


Near Far 


Figure 3.9. Experiment 4: Spatial Difference 


The room is prepared in the same manner as the experiment one with one virtual and 
one real cardboard box. Each iteration of the experiment features one real and one virtual 
box placed at two differing distances from the entrance to the room within the FOV of the 


participant. 
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The boxes are randomly set farther or nearer from the entrance by a ratio of 2:1 (the 
farther box is twice as far as the near box) (Figure 3.10). Each participant executes this 
experiment two times (Table 3.7). 


Table 3.5. Experiment Four Iterations 





Iteration # | Real Box Position | Virtual Box Position 


1 Near Far 
2 Far Near 


























Table 3.6. Experiment Four Iterations for the Real Box 





Iteration # | Real Box 1 Position | Real Box 2 Position 


1 Near Far 























Table 3.7. Experiment Four Iterations for the Virtual Box 





Iteration # | Virtual Box 1 Position | Virtual Box 2 Position 


1 Near Far 























For each iteration of the experiment four, specific data are recorded to capture the room 


and environment set up. For experiment four, this includes the following data: 


. Do the participants scan the room? Yes/No. 
. Which object do they visually assess first? Left/Right. 
. Time it takes them to move toward an object. 


. Location of virtual box: Near or Far. 


nA BB WN = 


. Location of real box: Near or Far. 


The data collected on each iteration of this experiment provides data for statistical 
analysis and highlighting information for future experiments. This experiment does not 
take into account that the farther box appears smaller to the participant as it is negligible 
given the constraints of the experiment room. Future work includes addressing this topic 
by using a longer room. Additional future work includes combinations with the boxes in 


different locations within the participants FOV, one visible and one not. 


a1 


Spatial Difference Experiment Room Layout Example 


Object 


Object 


Subject’s FOV 








Outside of FOV Outside of FOV 


Subject 





Figure 3.10. Spatial Difference Experiment Room Layout 


3.2 Questionnaire 

The purpose of the questionnaires is to capture key demographic data of each participant. 
This data facilitates statistical analysis to place participants into categories for comparison. 
Previous experience with AR is paramount to identify as those participants may have a 
reduced bias towards the virtual objects. This assumption is made based on Tversky and 
Kahneman’s availability heuristic [19]. 


A pre-test questionnaire is given upon completion of each experiment: 


. What is your sex? 
. What is your age? 
. Are you right or left-handed? 
Do you use corrective lenses? 


. Is English your first language? 


. If military, how many years of service do you have? 
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7. Do you have experience with search tasks or search patterns? 


11. 
12. 


. When was the last time you conducted this kind of task? 
. Do you have experience with augmented reality? If yes, please explain. 
10. 


What is your attitude towards augmented reality technology? (Likert scale 1-5, 1-hate 
it, 3-like it, 5- love it) 
How many hours did you sleep last night? 


Are you color blind? If so in what colors: 


The post-test questionnaire includes the following questions: 


1. 


Have you ever received any training or had experience in how to search or search 


patterns? If yes, what type and when? 


2. Did you see both boxes in the room? 


NYDN iss) 


\o © 


. Which object did you notice first when you entered the room (object on the right or 


the left)? 


. Which object did you approach first (object on the right or the left)? 
. Please explain why you think you picked that object first. 
. Given a choice, would you rather interact with all real objects or all rendered objects? 


. Given the task we asked you to conduct, what did you dislike about the environment 


and what did you like? 


. Do you think augmented reality could be used for more than just training? 
. How do you think augmented reality could be improved? 
10. 


Questions, Comments or Concerns about anything dealing with the experiment. 


The simulator sickness questionnaire (Appendices B and C) is provided to each partici- 


pant upon the completion of the experiment. This is provided to collect data on the effects 


upon the participant that the AR headset may have upon them. 


3.3 


Equipment 


The Magic Leap | headset with a Grip Totem controller and the Unity game engine were 
utilized to design the VE. The Magic Leap | AR-HMD software version was OS 0.98.01. 


Unity 5, version 2019.1.0f2, was used to create the VE and implement the box texture, 


material, color, lighting, and placement within the AR-HMD FOV for the wearer. Unity is 


a robust game engine that is offered free for personal and educational use. A paid version 
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is available for professional product developers. In order to develop the VE in Unity for 
the Magic Leap, the Magic Leap Unity Lumin SDK version 0.20.0.2019.320 was installed 
within Unity. Microsoft’s Visual Studio was used to edit and create C-sharp scripts for the 


“in game” manipulation of the box. 


A Hewlett-Packard desktop computer running Microsoft Windows 10 with a NVIDIA 
GeForce GTX 1080 Ti graphics card ran Unity and built the virtual environment. This 
computer setup met or exceeded the minimum system requirements for Magic Leap 1 


development with Unity. 


The physical cardboard box for all experiments is a brown UPS shipping box. The 
rendered box was scaled appropriately based on the experiment to match or scale the 


dimensions of the physical box. 


3.4 Procedures 

A randomization of the VE room layout is implemented with each variation of the 
experiment to ensure a random variable for statistical analysis. For example: (1) randomly 
changing the placement of the real or rendered box in the room; (2) randomly changing the 


size or color of the real or rendered box in the room. 


When participants arrive at the experiment site the following procedures are followed 
in the same order to reduce variability between participant knowledge, familiarity, and 


understanding of the experiment. 
Pre-Experiment: 


. Welcome each participant by the same researcher upon arrival. 

. Provide the informed consent to the participant for them to read, understand, and sign. 
. Provide the participant with the scripted instructions. 

. Provide the pre-test questionnaire to the participant. 


. Ensure proper fit of the familiarization Magic Leap | headset. 


NO” BW NY 


. Provide a 5 to 10-minute equipment familiarization to each participant with a second 
Magic Leap | headset. Limit each participant to no more than 10 minutes to reduce 


the likelihood of cyber sickness effects impacting their ability to participate in the 
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10. 


experiment. The “Create” app on Magic Leap | is the preferred application for 
familiarization. The “Create” application allows users to interact with virtual objects 
and adapt to viewing objects through the AR-HMD lens. Utilizing the same app 
across all experiments reduces variability of familiarization between participants. 
(Figure 3.11) 


. Ensure the participant understands the requirements asked of them by asking them to 


repeat back the instructions. 


. With the test Magic Leap 1 headset verify the virtual and physical room layout is 


correct. 


. Ensure proper fit of the test Magic Leap | headset and that the participant can see the 


remote in their hand. Prior to the next step take the remote back from the participant 
as it is not required for any iteration of the experiment. 
Re-iterate the experiment requirements to the participant, i.e., searching for a specific 


object (rubber duck) within the room. 


Experiment: 


1. 


Conduct experiment and record results. Ensure that the same researcher is recording 


these results. 


Post-Experiment: 


. Explain that the rubber duck did not exist, and the true nature of the experiment was 


to record human bias towards an object in a VE. 


. Ask the participant not to discuss the experiment with other participants waiting to 


conduct the experiment. 


. Provide the post-test questionnaire, simulator sickness questionnaire, document any 


feedback from the participant, and answer any questions. 


. Thank the participant for their time and cooperation. 


4] 





Figure 3.11. Magic Leap 1 Create App. Source: [50]. 


3.5 Demographics 
In order to reduce variation in participant demographics from Frisco’s research, every 


effort must be made to obtain military participants with ages between 18 and 42 years old 
and military ranks between E-2 and E-7, and O-1 to O3. 


3.6 Analysis Plan 


Test data is analyzed in the same manner as Frisco’s previous experiment [13]. A one- 
proportioned Z-test, a two-proportioned Z-test, a two-sample T-test, a Wilcoxon signed-rank 
test, and a Chi-Squared test are implemented and compared with previous data. 


For all experiments, record the demographic information and behavior of the participant 
within the VE. The below variables are to be recorded during each experiment. 
¢ Predictor Variables 
— Gender 
Years of Service (< 3, 3-6, >6) 


Search Task Experience (Yes or No) 


— AR Experience (Yes or No) 
Visual Gaze (Left or Right) 


Room Scan (Yes or No) 
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— Directed motion. Which object did the participant more towards first? (Real or 
Virtual) 
— Length of time till the participant moved within three feet of a box 
¢ Random Variables 
— Room Layout (Placement of the real and virtual box is randomized between 
right and left ) 
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CHAPTER 4: 


Discussion and Summary 





The model for experimentation laid out in this research is foundational to understand the 
numerous influences, biases, and ramifications of decisions present while an individual is 
using an AR headset. The technology available to the DoD in the realm of MR is expanding 
rapidly as the miniaturization of computer chips continues to advance and graphics pro- 
cessing becomes more efficient. These systems provide a medium for enhanced learning, 
cognition, and visual perception. The planners and decision makers in charge of the ac- 
quisition and employment of such devices must understand the aforementioned influences, 
biases, and ramifications to ensure the safety and mission success of warfighters equipped 
with this technology. The current lack of experience by service members with this tech- 
nology impacts the decision-making heuristics, outlined by Tversky and Kahneman, that 
impact the users decision-making [19]. As experience increases the understanding of how 
decisions are in these environments made will increase as well. The very technological 
nature of the equipment and increased connectedness is grounds for future work as we seek 
to understand the impacts of Aiken’s “Cyber Effect” within the constraints of an intercon- 
nected AR environment. The world around the user provides subconscious affordances 
which influence decisions subconsciously and must be taken into account during interface 
design, depiction of information, and representation of objects. This model for experimen- 
tation provides a pathway to better understand the many influences on human decision. 
AR provides a cost-effective method for training and simulating real-life environments and 
may one day soon be fielded to operational military units for use in tactical situations. The 
ability to increase situation awareness is undoubtedly present as these systems provide the 
means to communicate effectively between members of large teams [40]. However, it is 
crucial to understand the impacts the virtual environment will have on the user’s decision- 
making process. Potentially dangerous ramifications may present themselves without this 
understanding in life or death decisions on the battlefield of the not so far future. 
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4.1 Implications 

As the DoD continues to plan for the future force and the technology which will be 
incorporated, a holistic understanding by leaders and a detailed understanding by designers 
is required when purchasing and designing technology for use on the battlefield. Decisions 
are influenced by information relayed to warfighters, specifically information displayed in 
an AR environment. The implications of this research for designers is the understanding 
of the influences fidelity, color, size, and placement of virtual objects in the user’s FOV 
have on their decisions. Ensuring that both negative or positive biases are not introduced 
to the user ensures the user has the best opportunity to make the correct decision in a given 


situation. 


4.2 Limitations 


Limitations present themselves in three areas: people, hardware, and software. 


Due to the safety concerns surrounding the global pandemic, the author was unable to 
obtain the required permission to conduct the experiments with participants. One limitation 
on future work will be gathering a large enough sample size of participants to obtain 


quantifiable data. 


The Magic Leap 1 headset is limited by current hardware technology for FOV and 
graphics processing. This limitation is only partly overcome through in-depth object design 
and rendering with Unity. As technology progresses and the virtual environment is less 
distinguishable from the real, these experiments should be repeated in order to continue 
to understand the human selection bias in an AR environment. Some of these hardware 


limitations may be overcome through use of the new Hololens 2. 


Software limitations and individual graphics coding skills is an ever present limitation 


that is only reduced through technological advances and experience. 


4.3 Future Work and Recommendations 
Upon identifying and analyzing the results from the experiments in this research plan, 
research should be conducted in the following areas: Identifying and reducing the com- 


pounding bias from the fidelity experiment to the follow-on experiments; understanding the 
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impact of poor visual acuity in the periphery in a MR environment on decision-making. 
Comparative analysis of military users employing AR to assist their day to day tasks such 
as object identification, route planning, equipment maintenance, and threat identification. 
Data should be compared to non-AR users for performance increases and errors made while 


employing this technology. 


Future experimentation addressing novelty, specific to an AR environment, should be 
conducted using Frisco’s results as a control group and a new group of experienced AR 
users. This experimentation has the potential to identify the level of experience required to 


reduce the impacts of novelty. 


Upon the execution of the four experiments conducting new experiments that merge 
aspects of each experiment is grounds for new research i.e., an array of colored virtual 
objects, combining size and spatial variables, etc. Future research builds the framework to 
understand the reactions of a user wearing an AR-HMD while identifying virtual and real 


threats. 


Additional future research and experimentation is needed to understand a critical baseline 
question: What is the level of fidelity a rendered object must have to be considered nearly 
indistinguishable from the real object? This research will explore the visual perception of 
the user and the ability to render the virtual object given current technological constraints. 


Researching the impacts of the “Cyber Effect’ as the warfighter becomes more intercon- 
nected with tablets, sensors, and AR devices will be special importance to understand the 


psychological impacts upon the warfighter. 
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APPENDIX A: 
Unity Configurations 





To develop the AR environment for each experiment the Unity game engine was utilized 
to build the virtual environment. The Lumin software development kit plug-in is required 
for all Unity code development for the Magic Leap | headset. The author implemented a 
mesh cardboard box to generate the basic structure of the virtual cardboard boxes that are 
viewed through the Magic Leap 1 AR-HMD. The virtual boxes are based on the Unity 
Asset Store asset “Cardboard Boxes” [51]. The Unity user interface allowed the author to 
implement specific shaders and lighting settings to generate virtual boxes that are realistic 
when viewed through the Magic Leap 1. The “Placeable” C-Sharp script [52] was applied 
to each virtual box to allow interaction via the Magic Leap 1 hand-held controller. This 
script enables the movement of the virtual box in the virtual space in order to place it within 


the experiment room. 


An optional “Cursor” script was implemented to ease the pointing of the controller. 
When the script’s line renderer is enabled a laser-like beam is visible through the AR-HMD 
extruding from the controller. Current compiled applications of the experiment have this 
feature disabled. It can be re-enabled by future experiment executors by selecting the “Line 
Renderer” under the “Cursor” GameObject within the Unity user interface. Figure A.1 is a 
depiction of the laser-like beam in Unity. 
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Figure A.1. Pre-Compiled View of the “Laser” pointer” 


The virtual cardboard box implements the Unity Standard (Specular setup) shader on 
the “Cardboard Boxes” asset. The following shader maps were implemented. The albedo 
is mapped to a bitmap texture file that resembled cardboard with the alpha cutoff set at 0.5. 
The normal map is a matte cardboard texture from the Unity Asset store “Cardboard realistic 
boxes” asset package. The detail mask is a normal bitmap texture from the “Cardboard 
Boxes” asset package. Emission is selected on with real-time global illumination and a 1:1 
tiling which overlays the aforementioned maps over the box mesh. “Specular Highlights,” 
“Enable GPU Instancing,” and “Double Sided Global Illumination” are enabled within the 
shader. These settings result in a more realistic virtual box once compiled and viewed 
through the Magic Leap 1 headset. The experiment room layout is intended to have direct 
overhead lighting, enabling the “Double Sided Global Illumination,” paired with a virtual 
area baked light, mimics the overhead light affect. 
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Figure A.2. Pre-Compiled Cardboard Box 


All rendered colored boxes implement the Unity standard shader with “Emission,” “Spec- 
ular Highlights,” “Reflections,’, “Enable GPU Instancing,” and “Double Sided Global II- 
lumination” enabled. The following color values are set for their respective virtual boxes. 


Figure A.3 is a pre-compiled example of the boxes Unity. 


Bright Red: RGB Values: 255, 0, 0. Alpha: 255. Hexadecimal color value: FFOO00 
Dark Red: RGB Values: 128, 0,0. Alpha: 255. Hexadecimal color value: 800000 
Bright Green: RGB Values: 0, 255, 0. Alpha: 255. Hexadecimal color value: OOFFOO 
Dark Green: RGB Values: 0, 128, 0. Alpha: 255. Hexadecimal color value: 008000 
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Figure A.3. Pre-Compiled Colored Boxes 


The virtual box for the spatial and size difference experiments is the same as in Figure 
A.2 with the exception of scaling for the size difference. 


Additional scripts were modified and implemented from Rodney Degracia, a Unity code 
developer who writes scripts for Magic Leap 1 [52]. The “Controller” script works in con- 
junction with the aforementioned “Cursor” script and is attached to the Unity GameObject 
for the Magic Leap 1 controller object. The script enables basic controller functionality 
for the touch pad and buttons. This controller is used by the individual setting up the 
experiment, not by the experiment participant. 


The Magic Leap 1 Lumin SDK Spatial Mapper prefab was implemented enabling the 
boxes interact with real world objects when viewed through the Magic Leap | headset i.e., 


if the virtual box is placed through a wall only a portion would remain visible to the wearer. 


Specific Unity light settings were implemented to provide virtual light sources on the 
virtual boxes to best mirror the overhead lights of the experiment room. Baked Global 
Illumination with a realtime shadow RGB color of 35, 42, 58 is set along with a directional 
light source from the virtual skybox. Figure A.4 is the Unity light settings used for all the 


virtual environments. 
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Figure A.4. Pre-Compiled Unity Light Settings 
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APPENDIX B: 
Simulator Sickness Questionnaire 





No Date 


SIMULATOR SICKNESS QUESTIONNAIRE 
Kennedy, Lane, Berbaum, & Lilienthal (1993)*** 


Instructions : Circle how much cach symptom below is affecting you right now. 





























1. General discomfort None Slight Moderate Severe 
2. Fatigue None Slight Moderate ver 

3. Headache None Slight Moderate Severe 
4. Eye strain None Slight Moderate ver 

5. Difficulty focusing None light Moderate Severe 
6. Salivation increasing None Slight Moderate Severe 
7. Sweating None Slight Moderat Severe 
8. Nausea None Slight Moderate Severe 
9. Difficulty concentrating None Slight Moderate Severe 
10. « Fullness of the Head » None Slight Moderate Severe 
11. Blurred vision None Slight Moderate Severe 
12. Dizziness with eyes open None Slight Moderate Severe 
13. Dizziness with eyes closed None Slight Moderate Severe 
14. *Vertigo None Slight Moderate Severe 
15. **Stomach awareness None Slight Moderate Severe 
16. Burping None Slight Moderate Severe 





* Vertigo is experienced as loss of orientation with respect to vertical upright. 


** Stomach awareness is usually used to indicate a feeling of discomfort which is just short of 
nausea. 


Last version ; March 2013 
***Original version : Kennedy, R.S., Lane, N.E., Berbaum, K.S., & Lilienthal, M.G. (1993). Simulator Sickness 


Questionnaire: An enhanced method for quantifying simulator sickness. Jnternational Journal of Aviation Psychology, 
3(3), 203-220. 
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APPENDIX C: 
Simulator Sickness Scoring 





This content is adapted from cybersickness.org 
(http://www.cybersickness.org/Simulator_Sickness_Questionnaire.htm). 


A brief explanation of the Simulator 
Sickness Questionnaire (SSQ) 


Simulator Sickness Questionnaire was developed by Kennedy and his colleagues in 1993 (Kennedy et al., 1993). 
They used over 1000 sets of previous data and through some analysis, they came up with a list of 27 symptoms 
which are commonly experienced by users of virtual reality systems. Each item is rated with the scale from none, 
slight, moderate to severe. Through some calculations, four representative scores can be found (Appendix B). 
Nausea-related subscore (N), Oculomotor-related subscore (O), Disorientation-related subscore (D) are the scores 
for the symptoms for the specific aspects. Total Score (TS) is the score representing the overall severity of 
cybersickness experienced by the users of virtual reality systems. Simulator Sickness Questionnaire is a widely 
applied measurement tool in research studying simulator sickness and cybersickness. 


e Pre-exposure Simulator Sickness Questionnaire >>download the questionnaire (/docs/Pre.doc) (.doc file) 
e Post-exposure Simulator Sickness Questionnaire >>download the questionnaire (/docs/Post.doc) (.doc file) 


The calculations in the Simulator Sickness Questionnaire 


None = 0 
Slight = 1 
Moderate = 2 
Severe = 3 
Weights for Symptoms 
Symptoms Nausea Oculomotor Disorientation 
General discomfort 1 1 
Fatigue 1 
Headache 1 
Eye strain 1 
Difficulty focusing 1 1 
https://www.twentymilliseconds.com/html/ssq-scoring.html Page 1 of 2 
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Increased salivation 1 


Sweating | 

Nausea 1 1 
Difficulty concentrating 1 1 

Fullness of head 1 
Blurred vision 1 1 
Dizzy (eyes open) 1 
Dizzy (eyes closed) 1 
Vertigo 1 
Stomach awareness 1 

Burping 1 

Total* [1] [2] [3] 
Score 


Nausea = [1] ’ 9.54 
Oculomotor = [2] ° 7.58 
Disorientation = [3] © 13.92 


Total Score = ([1] + [2] + [3]) *3.74 





* Total is the sum obtained by adding the symptoms scores. Omitted scores 
are zero 


https://www.twentymilliseconds.com/html/ssq-scoring.html 
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APPENDIX D: 
Cursor C Sharp Code 





// Copyright 2018 Rodney Degracia 

// MIT License: 

// Permission is hereby granted, free of charge, to any person 

// obtaining a copy of this software and associated documentation 

// files (the "Software"), to deal in the Software without 

// restriction, including without limitation the rights to use, 

// copy, modify, merge, publish, distribute, sublicense, and/or 

// sell copies of the Software, and to permit persons to whom the 

// Software is furnished to do so, subject to the following conditions: 
// The above copyright notice and this permission notice shall be 

// included in all copies or substantial portions of the Software. 

// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 

// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES 

// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 

// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 
// BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
// ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 
// CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 

// THE SOFTWARE. 


using UnityEngine; 
using UnityEngine.Events; 
using System.Collections; 


using UnityEngine.XR.MagicLeap; 


using Prestige; 


59 


// Represents the behavior of the Cursor, which also has a 

// LineRenderer 

// This script should be attached to the GameObject which represents 
// the "end" of the picker "ray". 

// Works in conjunction with the InputController script, which should 
// be attached to the GameObject that 

// represents the Magic Leap controller 


public class Cursor : MonoBehaviour 
{ 
[SerializeField] 


InputController inputController = null; 


public uint Width = 1; 

public uint Height = 1; 

public float HorizontalFovDegrees; 

public bool CollideWithUnobserved = false; 
public float defaultDistance = 9.0F; 


public delegate void CursorMove(Ray controllerRay, 
Transform cursorTransform, RaycastHit? raycast); 

public delegate void CursorHover(GameObject gameObject, 
Transform cursorTransform, RaycastHit raycastHit) ; 


public delegate void CursorStopHover(GameObject gameObject) ; 


public static event CursorMove OnCursorMove; 
public static event CursorHover OnCursorHover; 


public static event CursorStopHover OnCursorStopHover ; 


private WorldRaysManager worldRaysManager = null; 
private Renderer _renderer; 
private LineRenderer lineRenderer; 


private Transform adjustedCursorTransform; 
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private GameObject hoveredGameObject = null; 


protected Color color; 
protected bool scaleWhenClose = true; 
protected bool hit; 


protected WorldRaysManager GetWorldRaysManager() 
{ 


return worldRaysManager; 


} 


protected LineRenderer GetLineRenderer() 


i 


return lineRenderer; 


} 


public GameObject GetHoveredGameObject() 
{ 


return hoveredGameObject; 


i 


public Transform GetAdjustedCursorTransform() 


{ 


return adjustedCursorTransform; 


} 
private void Awake() 
{ 


Hashtable options = new HashtableQ; 


Debug.Assert(inputController != null, 


"inputControllerBehavior should be configured in the Inspector"); 
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options["Width"] = Width; 

options["Height"] = Height; 

options ["HorizontalFovDegrees"] = HorizontalFovDegrees; 
options["CollideWithUnobserved"] = CollideWithUnobserved; 


worldRaysManager = new WorldRaysManager (options) ; 


_renderer = GetComponent<Renderer>() ; 


lineRenderer = GetComponent<LineRenderer>() ; 
lineRenderer.startWidth = 0.01f; 
lineRenderer.endWidth = 0.01f; 

$ 


void StartQ 


{ 
worldRaysManager.Start (WorldRaysCallback) ; 


// Update is called once per frame 
void Update() 
{ 


worldRaysManager .Update(inputController.transform.position, 


inputController.transform. forward, inputController.transform.up) ; 


private void OnDestroy() 


{ 
worldRaysManager.Stop(); 


Z 
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virtual public void WorldRaysCallback(MLWorldRays. 
MLWorldRaycastResultState 
state, RaycastHit result, float confidence) 
{ 
Vector3 rayCastOrigin = inputController.transform. position; 
Vector3 rayCastDirection = inputController.transform. forward; 


if (state != MLWorldRays.MLWorldRaycastResultState.RequestFailed && 
state != MLWorldRays.MLWorldRaycastResultState.NoCollision) 


{ 
// Update the cursor position and normal. 
transform.position = result.point; 
transform. LookAt (result.normal + result.point) ; 
transform. localScale = Vector3.one; 
// Set the color to yellow if the hit is unobserved. 
_renderer.material.color = (state == 
MLWorldRays.MLWorldRaycastResultState.HitObserved) ? 
color : Color.yellow; 
if (scaleWhenClose) 
{ 
// Check the hit distance. 
if Cresult.distance < 1.0f) 
{ 
// Apply a downward scale to the cursor. 
transform.localScale = new Vector3(result.distance, 
result.distance, result.distance) ; 
} 
$ 
hit = true; 
} 
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else 
{ 
// Update the cursor position and normal. 
transform.position = (rayCastOrigin + 
(rayCastDirection * defaultDistance)); 
transform. LookAt (rayCastOrigin) ; 


transform. localScale = Vector3.one; 


_renderer.material.color = Color.red; 


hit = false; 


lineRenderer.positionCount = 2; 
lineRenderer.SetPosition(9, rayCastOrigin) ; 
lineRenderer.SetPosition(1, transform. position) ; 


lineRenderer.useWorldSpace = true; 


bool hitWorldMesh = (hit == true); 
bool outsideWorldMesh = (hit == false); 


RaycastHit raycastHit; 

System.Int32 defaultLayer = 1; // 0000 0001 

bool didHitGameObject = (Physics.Raycast (rayCastOrigin, 
rayCastDirection, 
out raycastHit, Vector3.Distance(rayCastOrigin, 
transform. position), 
defaultLayer)); 

Ray controllerRay = new Ray(rayCastOrigin, rayCastDirection) ; 


if (didHitGameObject) 
{ 
Debug.Log("Hit GameObject") ; 
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// Update the cursor position and normal. 
transform.position = raycastHit.point; 
transform. LookAt (raycastHit.normal + raycastHit.point) ; 
transform. localScale = Vector3.one; 


adjustedCursorTransform = transform; 


// Adjust ray to end at the inGamHit 


lineRenderer.SetPosition(1, raycastHit.point) ; 


// Set the color to yellow if the hit is unobserved. 
_renderer.material.color = (state == 
MLWorldRays.MLWorldRaycastResultState.HitObserved) ? 


color : Color.yellow; 


if (scaleWhenClose) 


{ 
// Check the hit distance. 
if (raycastHit.distance < 1.0) 
{ 
// Apply a downward scale to the cursor. 
transform.localScale = new Vector3(raycastHit.distance, 
raycastHit.distance, raycastHit.distance) ; 
} 
$ 


bool newObject = (hoveredGameObject == null); 

bool differentObject = (hoveredGameObject != null && 
raycastHit.collider.gameObject.GetInstanceID() != 
this. hoveredGameObject.GetInstanceID()) ; 


if (newObject) 
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hoveredGameObject = raycastHit.collider.gameObject; 
OnCursorHover (hoveredGameObject, transform, raycastHit) ; 


lineRenderer.material.color = Color.green; 


} 

else if (differentObject) 

{ 
OnCursorStopHover (hoveredGameObject) ; 
hoveredGameObject = raycastHit.collider.gameObject; 
OnCursorHover (hoveredGameObject, transform, raycastHit) ; 
lineRenderer.material.color = Color.green; 

} 

else 

{ 
// Same Object 
: // do nothing 

} 


OnCursorMove(controllerRay, adjustedCursorTransform, raycastHit) ; 


i 


else 


{ 
if (hitWorldMesh) 


{ 
Debug.Log("Hit world mesh"); 


lineRenderer.material.color = Color.yellow; 


if (this.hoveredGameObject != null) 
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OnCursorStopHover (hoveredGameObject) ; 
hoveredGameObject = null; 


OnCursorMove(controllerRay, transform, result); 


$ 
else if CoutsideWorldMesh) 
{ 
Debug.Log("Did not hit world mesh or game object"); 


lineRenderer.material.color = Color.red; 


adjustedCursorTransform = transform; 
adjustedCursorTransform.position = rayCastOrigin + 
(rayCastDirection.normalized * 2); 


OnCursorMove(controllerRay, adjustedCursorTransform, null); 


if (this. hoveredGameObject != null) 

{ 
OnCursorStopHover (hoveredGameObject) ; 
hoveredGameObject = null; 
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APPENDIX E: 
Input Controller C Sharp Code 





// Copyright 2018 Rodney Degracia 

// MIT License: 

// Permission is hereby granted, free of charge, to any person 

// obtaining a copy of this software and associated documentation 

// files (the "Software"), to deal in the Software without 

// restriction, including without limitation the rights to use, 

// copy, modify, merge, publish, distribute, sublicense, and/or 

// sell copies of the Software, and to permit persons to whom the 

// Software is furnished to do so, subject to the following conditions: 
// The above copyright notice and this permission notice shall be 

// included in all copies or substantial portions of the Software. 

// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 

// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES 

// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 

// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 
// BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
// ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 
// CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 

// THE SOFTWARE. 


using UnityEngine; 

using UnityEngine.Events; 
using System.Collections; 
using UnityEngine.Assertions; 


using UnityEngine.XR.MagicLeap; 


using Prestige; 
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// Controller Behavior should be attached to the gameObject that 
// represents the Magic Leap Controller. This script works in 

// conjunction with the Cursor script, which should be attached to 
// the GameObject which represents the Cursor 

// (end of the picker "ray"). 


public class InputController : MonoBehaviour 
{ 
[SerializeField] 


Cursor cursor = null; 


public delegate void TriggerDown(byte controllerId, float value, 
GameObject gameObject, Transform cursorTransform) ; 
public delegate void TriggerUp(byte controllerId, float value, 


GameObject gameObject, Transform cursorTransform) ; 


public static event TriggerDown OnTriggerDown; 


public static event TriggerUp OnTriggerUp; 


public delegate void TouchpadGestureStart(byte controllerId, 
MLInputControllerTouchpadGesture gesture, 


Cursor cursor); 


public delegate void TouchpadGestureEnd(byte controllerId, 
MLInputControllerTouchpadGesture gesture, 


Cursor cursor); 


public static event TouchpadGestureEnd OnTouchpadGestureEnd; 


public static event TouchpadGestureStart OnTouchpadGestureStart; 


private Prestige.InputController inputController; 
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protected Prestige. InputController GetInputController() 
{ 


return inputController; 


virtual public void AwakeQ) 
{ 
Assert.raiseExceptions = true; // Debugging 
inputController = 
new Prestige. InputController(Prestige.DeviceType.ControllerFirst) ; 
inputController.StartQ; 
$ 


virtual public void OnDestroy() 
{ 
inputController.StopQ; 


virtual public void Update() 


{ 

if CinputController != null) 

{ 

MLInputController mlInputController = inputController. 
GetMLInputControllerQ ; 


Assert.IsTrue(mlInputController != null); 
if CmlInputController != null) 
{ 


transform.position = mlInputController.Position; 


transform.rotation = mlInputController.Orientation; 
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private void OnEnable() 


{ 
if CinputController == null) 
{ 
inputController = new 
Prestige. InputController(Prestige.DeviceType.ControllerFirst) ; 
inputController.StartQ ; 
} 


inputController.RegisterTriggerUpHandler (OnTriggerUpHandler) ; 


inputController.RegisterTriggerDownHandler(OnTriggerDownHand1ler) ; 


inputController.RegisterTouchpadGestureStartHandler 
COnTouchpadGestureStartHandler) ; 

inputController.RegisterTouchpadGestureEndHandler 
(OnTouchpadGestureEndHandler) ; 


private void OnDisableQ 

{ 

if CinputController != null) 

{ 

inputController.UnregisterTriggerUpHandler (OnTriggerUpHandler) ; 


inputController.UnregisterTriggerDownHandler(OnTriggerDownHandler) ; 


inputController.UnregisterTouchpadGestureStartHandler 
COnTouchpadGestureStartHandler) ; 

inputController.UnregisterTouchpadGestureEndHandler 
COnTouchpadGestureEndHandler) ; 


2. 


virtual public void OnTriggerDownHandler(byte controllerId, 
float triggerValue) 


{ 
if COnTriggerDown == null) 
{ 
return; 
$ 


GameObject hoveredGameObject = cursor.GetHoveredGameObject() ; 
Transform cursorTransform = cursor.GetAdjustedCursorTransform() ; 


if ChoveredGameObject == null) 

{ 

OnTriggerDown(controllerId, triggerValue, null, cursorTransform) ; 
+ else 

{ 

OnTriggerDown(controllerId, triggerValue, hoveredGameObject, 


cursorTransform) ; 


virtual public void OnTriggerUpHandler(byte controllerld, 
float triggerValue) 


{ 
if (COnTriggerUp == null) 
{ 
return; 
$ 
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GameObject hoveredGameObject = cursor.GetHoveredGameObject() ; 


Transform cursorTransform = cursor.GetAdjustedCursorTransform() ; 
OnTriggerUp(controllerId, triggerValue, hoveredGameObject, 


cursorTransform) ; 


virtual public void OnTouchpadGestureStartHandler(byte controllerId, 
MLInputControllerTouchpadGesture gesture) 


{ 
if COnTouchpadGestureStart != null) 
{ 
OnTouchpadGestureStart(controllerId, gesture, cursor); 
$ 
$ 


virtual public void OnTouchpadGestureEndHandler(byte controllerId, 
MLInputControllerTouchpadGesture gesture) 


{ 
if COnTouchpadGestureEnd != null) 
{ 
OnTouchpadGestureEnd(controllerId, gesture, cursor); 
$ 
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APPENDIX F: 
Meshing C Sharp Code 





// Copyright 2018 Rodney Degracia 

// MIT License: 

// Permission is hereby granted, free of charge, to any person 

// obtaining a copy of this software and associated documentation 

// files (the "Software"), to deal in the Software without 

// restriction, including without limitation the rights to use, 

// copy, modify, merge, publish, distribute, sublicense, and/or 

// sell copies of the Software, and to permit persons to whom the 

// Software is furnished to do so, subject to the following conditions: 
// The above copyright notice and this permission notice shall be 

// included in all copies or substantial portions of the Software. 

// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 

// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES 

// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 

// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 
// BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
// ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 
// CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 

// THE SOFTWARE. 

using System.Collections; 

using System.Collections.Generic; 

using UnityEngine; 


using UnityEngine.XR.MagicLeap; 


public class Meshing : MonoBehaviour { 
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public Material material; 


// Use this for initialization 
void Start Q) f{ 


// Update is called once per frame 
void Update () { 
UpdateMeshMaterialQ(); 


public void UpdateMeshMaterial(Q 


{ 


// Loop over all the child mesh nodes created by 

// MLSpatialMapper script 

for (int i = 0; i < transform.childCount; i++) 

{ 

// Get the child gameObject 

GameObject gameObject = transform.GetChild(i).gameObject; 

// Get the meshRenderer component 

MeshRenderer meshRenderer = 
gameObject.GetComponent<MeshRenderer>() ; 

meshRenderer.material = material; 


7 
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APPENDIX G: 
Placeable C Sharp Code 





// Copyright 2018 Rodney Degracia 

// MIT License: 

// Permission is hereby granted, free of charge, to any person 

// obtaining a copy of this software and associated documentation 

// files (the "Software"), to deal in the Software without 

// restriction, including without limitation the rights to use, 

// copy, modify, merge, publish, distribute, sublicense, and/or 

// sell copies of the Software, and to permit persons to whom the 

// Software is furnished to do so, subject to the following conditions: 
// The above copyright notice and this permission notice shall be 

// included in all copies or substantial portions of the Software. 

// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 

// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES 

// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 

// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 
// BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
// ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 
// CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 

// THE SOFTWARE. 


using System.Collections; 
using System.Collections.Generic; 
using UnityEngine; 


using UnityEngine.XR.MagicLeap; 


// Add this script to any GameObject that is to be pickable 
// and placeable. 


TT 


// Note: This script should be used with GameObjects that do not 
// have a Rigidbody, since this script only modifies the 
// GameObject transform.position 


public class Placeable : MonoBehaviour 


{ 

public enum PlaceableState 

{ 
READY, 
NOSELECTED, 
ELIGIBLE_FOR_SELECTION, 
NOT_ELIGIBLE_FOR_SELECTION, 
SELECTED 

} 

[SerializeField] 


public Material onHoverMaterial; 


[SerializeField] 


public Material onSelectedMaterial; 


private Material saveMaterial; 

private Ray controllerRay; 

private float clampDistance; 

protected PlaceableState placeableState; 


protected Ray GetControllerRay() 
{ 


return controllerRay; 


protected float GetClampDistance() 
{ 
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return clampDistance; 


private void Awake() 
{ 
clampDistance = 0.0F; 
placeableState = PlaceableState.READY; 


void StartQ 
1 
saveMaterial = this.gameObject.GetComponent<Renderer>(). 


material; 


// Update is called once per frame 
void Update() 

{ 

$ 


protected void OnEnable() 


{ 
Cursor.OnCursorMove += OnCursorMove; 
Cursor .OnCursorHover += OnCursorHover; 
Cursor.OnCursorStopHover += OnCursorStopHover ; 
InputController.OnTriggerDown += OnTriggerDown; 
InputController.OnTriggerUp += OnTriggerUp; 

$ 


protected void OnDisable() 
{ 


InputController.OnTriggerDown -= OnTriggerDown; 
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InputController.OnTriggerUp -= OnTriggerUp; 


Cursor.OnCursorMove -= OnCursorMove; 
Cursor.OnCursorHover -= OnCursorHover; 
Cursor.OnCursorStopHover -= OnCursorStopHover ; 


// We use a statemachine, since events may occur asynchronously, 


// to help maintain state. 


protected void ExecuteStateMachine(PlaceableState sm) 


{ 


switch (sm) 


t 


case PlaceableState.NOSELECTED: 


{ 
if @placeableState != PlaceableState.SELECTED) 


rf 


return; 


} 


clampDistance = 9.0F; 


// reset the clamp, because the Gameobject is no longer selected 


placeableState = PlaceableState.READY; 


this.gameObject .GetComponent<Renderer>() .material = 


saveMaterial; 


break; 


t 
case PlaceableState.NOT_ELIGIBLE_FOR_SELECTION: 


{ 
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if C@placeableState != 
PlaceableState.ELIGIBLE_FOR_SELECTION) 


return; 
} 
placeableState = PlaceableState.READY; 
break; 
$ 
case PlaceableState.ELIGIBLE_FOR_SELECTION: 
{ 
if QplaceableState != PlaceableState.READY) 
{ 
return; 
$ 
placeableState = PlaceableState.ELIGIBLE_FOR_SELECTION; 
break; 
$ 
case PlaceableState.SELECTED: 
{ 
if (placeableState != 
PlaceableState.ELIGIBLE_FOR_SELECTION) 


return; 


} 
placeableState = PlaceableState.SELECTED; 


this.gameObject .GetComponent<Renderer>().material = 


onSelectedMaterial; 


break; 


} 
case PlaceableState.READY: 


{ 
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break; 
$ 
default: 
break; 


public void OnCursorMove(Ray controllerRay, 
Transform cursorTransform, RaycastHit? raycast) 

{ 

if (@placeableState == PlaceableState.SELECTED) 

{ 

/// 

/// Calculate the distance of the original controller ray, 

///when the game object was 

/// first selected 

ass 

var heading = GetComponent<Renderer>().bounds.center - 
this.controllerRay.origin; 


var distance = heading.magnitude; 


// Clamp the distance so that the distance 
// from the InputController to the GameObject 
// does not change while the GameObject is selected. 
if (Mathf.Abs(clampDistance - 0) < float.Epsilon) 
{ 
clampDistance = distance; 
$ 
// Move the game Object to a position on the Ray, 
// at the clamped distance 
Vector3 position = controllerRay.GetPoint(clampDistance) ; 
this.transform.position = position; 


} 
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if @laceableState == PlaceableState.NOSELECTED) 
{ 


this.controllerRay = controllerRay; 


public void OnCursorHover(GameObject gameObject, 
Transform cursorTransform, RaycastHit raycastHit) 

{ 

eA 

/// Return if we are not the gameObject that is being 

/// hovered by the Cursor 


/// 
if (this.gameObject.GetInstanceID(Q) != gameObject.GetInstanceID()) 


{ 


return; 


this.gameObject.GetComponent<Renderer>().material = 


onHoverMaterial; 


ExecuteStateMachine(PlaceableState.ELIGIBLE_FOR_SELECTION) ; 


public void OnCursorStopHover (GameObject gameObject) 


{ 
ExecuteStateMachine(PlaceableState.NOT_ELIGIBLE_FOR_SELECTION) ; 


this.gameObject.GetComponent<Renderer>() .material = saveMaterial; 


} 


public void OnTriggerDown(byte controllerId, float value, 
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GameObject gameObject, Transform cursorTransform) 


ExecuteStateMachine (PlaceableState.SELECTED) ; 


public void OnTriggerUp(byte controllerId, float value, 


GameObject gameObject, Transform cursorTransform) 


ExecuteStateMachine(PlaceableState.NOSELECTED) ; 
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APPENDIX H: 
Prestige C Sharp Code 





// Copyright 2018 Rodney Degracia 

// MIT License: 

// Permission is hereby granted, free of charge, to any person 

// obtaining a copy of this software and associated documentation 

// files (the "Software"), to deal in the Software without 

// restriction, including without limitation the rights to use, 

// copy, modify, merge, publish, distribute, sublicense, and/or 

// sell copies of the Software, and to permit persons to whom the 

// Software is furnished to do so, subject to the following conditions: 
// The above copyright notice and this permission notice shall be 

// included in all copies or substantial portions of the Software. 

// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 

// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES 

// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 

// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS 
// BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
// ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 
// CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 

// THE SOFTWARE. 


using System; 

using System.Collections.Generic; 
using UnityEngine; 

using UnityEngine.Events; 

using System.Collections; 

using UnityEngine.XR.MagicLeap; 


using UnityEngine.Assertions; 
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// Prestige 1S a namespace that wraps the Magic Leap SDK objects 


// for easier programming by keeping as much of the Magic Leap 


// SDK objects within the Prestige namespace, thereby keeping the 


// Magic Leap specific code from polluting the Application code. 


// By keeping the Application code as Unity-specific as possible, 


// Unity developers should have an easier time porting Unity 


// specific codebases to work with Magic Leap controller input 


// via Prestige. 


namespace Prestige 


{ 


//Possible devices that may be used as Controllers 


//Interestingly, the left hand controller is considered by 


//Magic Leap to be the first controller 


[Flags] 


public enum DeviceType : int 


{ 


t 

td. 
// 
// 
+4. 
// 


MobileApp = 1 << 9Q, 
ControllerFirst = 1 << 1, // Left 
ControllerSecond = 1 << 2, // Right 


InputControllerManager manages the possible devices 
that may be used as Magic Leap Controllers. 

This class manages the connection and disconnection of 
Magic Leap controllers, via 

a list of MLInputControllers. 


public class InputControllerManager 


{ 


private DeviceType deviceTypesAllowed = (DeviceType)~9; 
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private List<MLInputController> inputControllers = 
new List<MLInputController>Q ; 


private MLInputConfiguration inputConfiguration; 


protected MLInputConfiguration GetMLInputConfigurationQ 
{ 


return inputConfiguration; 


protected List<MLInputController> GetInputControllers() 
{ 


return inputControllers; 


public InputControllerManager () 

{ 

inputConfiguration = 

new MLInputConfiguration( 
MLInputConfiguration.DEFAULT_TRIGGER_DOWN_THRESHOLD , 
MLInputConfiguration.DEFAULT_TRIGGER_UP_THRESHOLD , 
true); 


public InputControllerManager ( 


MLInputConfiguration inputConfiguration) 


this.inputConfiguration = inputConfiguration; 


virtual public void StartQ 
{ 
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if (!MLInput.IsStarted) 

{ 

if (MagicLeapDevice.IsReady() ) 

{ 

MLResult result = MLInput.Start(inputConfiguration) ; 

if (result.IsOk == true) 

{ 

MLInput.OnControllerConnected += 
HandleOnControllerConnected; 

MLInput.OnControllerDisconnected += 


HandleOnControllerDisconnected; 


MLInputController leftController = 
MLInput.GetController(MLInput .Hand.Left) ; 
AddInputController(leftController) ; 


MLInputController rightController = 
MLInput.GetController(MLInput.Hand.Right) ; 
AddInputController(rightController) ; 


MLInputController mobileController = 
MLInput.GetController(0); // Mobile 

AddInputController(mobileController) ; 

$ 

$ 


virtual public void AddInputController( 
MLInputController newController) 
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if (IsDeviceAllowed(newController)) 

{ 

if CinputControllers.Exists((device) => 
device.Id == newController.Id)) 

{ 

Debug. LogWarning (string. Format ( 
"Connected controller with id {9} already connected.", 
newController)); 

return; 


$ 
inputControllers.Add(newController) ; 
$ 
} 


virtual public void StopQ 


{ 
if (MLInput.IsStarted) 
{ 
MLInput.StopQ ; 
MLInput.OnControllerConnected += 
HandleOnControllerConnected; 
MLInput.OnControllerDisconnected += 
HandleOnControllerDisconnected; 
i: 
:: 


public MLInputController GetFirstController(Q 


{ 
Assert .IsNotNull(CinputControllers) ; 


foreach (var inputController in inputControllers) 
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bool isLeftHandController = (inputController.Type == 
MLInputControllerType.Control && inputController.Hand 
== MLInput.Hand.Left); 
if (isLeftHandController) 
{ 


return inputController; 


} 


return null; 


public MLInputController GetSecondControllerQ 
{ 
Assert.IsNotNull(CinputControllers) ; 
foreach (var inputController in inputControllers) 
{ 
bool isRightHandController = (inputController.Type == 
MLInputControllerType.Control && inputController.Hand 
== MLInput.Hand.Right) ; 
if (isRightHandController) 
{ 


return inputController; 


} 


return null; 


public MLInputController GetMobileAppController( 
{ 
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foreach (var inputController in inputControllers) 
{ 
bool isMobileApp = CinputController.Type 
== MLInputControllerType.MobileApp) ; 
if CisMobileApp) 
{ 


return inputController; 


} 


return null; 


protected bool IsDeviceAllowed(MLInputController inputController) 
{ 


if CinputController == null || !inputController.Connected) 


{ 


return false; 


bool isMobileApp = ((deviceTypesAllowed & DeviceType.MobileApp) 
!= 0 && inputController.Type 
== MLInputControllerType.MobileApp) ; 
bool isLeftHandController = ( 
(deviceTypesAllowed & DeviceType.ControllerFirst) 
!= 0 && inputController.Type 
== MLInputControllerType.Control && inputController.Hand 
== MLInput.Hand.Left); 
bool isRightHandController = ( 
(deviceTypesAllowed & DeviceType.ControllerSecond) 
!= 0 && inputController.Type 
== MLInputControllerType.Control && inputController.Hand 
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== MLInput.Hand.Right) ; 


return (isMobileApp | | 
isLeftHandController | | 
isRightHandController) ; 


virtual protected void HandleOnControllerConnected( 
byte controllerId) 

{ 

MLInputController newController = MLInput.GetController( 
controllerId); 

Assert.IsNotNull (newController) ; 


if (IsDeviceAllowed(newController)) 
{ 
if CinputControllers.Exists( 
(device) => device.Id == controllerId)) 
{ 
Debug. LogWarning (string. Format ( 
"Connected controller with id {9} already connected.", 
controllerId)); 
return; 


} 


inputControllers.Add(newController) ; 
$ 
} 


virtual protected void HandleOnControllerDisconnected( 
byte controllerId) 
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inputControllers.RemoveAl11( 


(device) => device.Id == controllerId); 


// InputController wraps the MLInput of an MLInputController 
// and provides access to the associated MLInputController, 
// if desired. 


public class InputController 


{ 


private DeviceType deviceType; 


private static InputControllerManager controllerManager; 


protected DeviceType GetDeviceType() 
{ 


return deviceType; 


protected static InputControllerManager GetControllerManager () 


1 


return controllerManager; 


public InputController() 
{ 


if CcontrollerManager == null) 


{ 


controllerManager = new InputControllerManagerQ() ; 
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public InputController(MLInputConfiguration inputConfiguration) 


{ 


if C(controllerManager == null) 


{ 


controllerManager = new InputControllerManager ( 


inputConfiguration) ; 


public InputController(DeviceType deviceType) 


{ 
if CcontrollerManager == null) 
{ 
controllerManager = new InputControllerManagerQ() ; 
$ 
this.deviceType = deviceType; 
$ 


public void StartQ 
{ 


Assert.IsTrue(controllerManager != null); 


controllerManager.Start() ; 


public void Stop( 
{ 


controllerManager.Stop(); 
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public void RegisterTouchpadGestureStartHandler ( 
MLInput.ControllerTouchpadGestureDelegate callback) 


MLInput.OnControllerTouchpadGestureStart += callback; 


private void MLInput_OnControllerTouchpadGestureStart ( 


byte controllerId, 


MLInputControllerTouchpadGesture touchpadGesture) 


throw new NotImplementedException() ; 


public void RegisterTouchpadGestureEndHandler ( 


MLInput.ControllerTouchpadGestureDelegate callback) 


MLInput.OnControllerTouchpadGestureStart += callback; 


public void UnregisterTouchpadGestureStartHandler ( 


MLInput.ControllerTouchpadGestureDelegate callback) 


MLInput.OnControllerTouchpadGestureStart -= callback; 


public void UnregisterTouchpadGestureEndHandler( 


MLInput.ControllerTouchpadGestureDelegate callback) 


MLInput.OnControllerTouchpadGestureStart -= callback; 
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public void 


MLInput. 


MLInput. 


public void 


MLInput. 


MLInput. 


public void 


MLInput. 


MLInput. 


public void 


MLInput. 


MLInput. 


RegisterTriggerDownHandler( 
TriggerDelegate callback) 


OnTriggerDown += callback; 


RegisterTriggerUpHandler ( 
TriggerDelegate callback) 


OnTriggerUp += callback; 


UnregisterTriggerDownHand1ler ( 


TriggerDelegate callback) 


OnTriggerDown += callback; 


UnregisterTriggerUpHandler( 
TriggerDelegate callback) 


OnTriggerUp += callback; 


public MLInputController GetMLInputControllerQ 


{ 


Assert.IsNotNull CcontrollerManager) ; 


switch (deviceType) 
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{ 
case DeviceType.MobileApp: 
{ 
return controllerManager.GetMobileAppController(); 
$ 
case DeviceType.ControllerFirst: 
{ 
return controllerManager.GetFirstController(); 
} 
case DeviceType.ControllerSecond: 
{ 
return controllerManager.GetSecondControllerQ; 
$ 
default: 
Assert.IsTrue(false) ; // Should not assert 


return null; 


Wraps MLWorldRays.QueryParams() behavior 

* and calls back to the 

Application code via worldRaysCallback. 

#/ 

public class WorldRaysManager 

{ 

public delegate void WorldRaysCallback( 
MLWorldRays.MLWorldRaycastResultState state, 
RaycastHit result, float confidence) ; 
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private WorldRaysCallback worldRaysCallback; 


private MLWorldRays.QueryParams queryParams = 
new MLWorldRays.QueryParams () ; 


private Func<MLWorldRays.MLWorldRaycastResultState, 
RaycastHit, float> callback; 


protected MLWorldRays.QueryParams QueryParams() 
{ 


return queryParams; 


public WorldRaysManager () 
{ 


queryParams = new MLWorldRays.QueryParams() ; 


public WorldRaysManager(Hashtable options) 
{ 


queryParams = new MLWorldRays.QueryParams() ; 


queryParams.Width = Cuint)options["Width"]; 
queryParams.Height = (uint)options["Height"]; 
queryParams.HorizontalFovDegrees = 

(float) options ["HorizontalFovDegrees" ]; 
queryParams.CollideWithUnobserved = 

(bool) options ["CollideWithUnobserved" ] ; 


virtual public void Start( 
WorldRaysCallback worldRaysCallback) 
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if (MagicLeapDevice.IsReady()) 


{ 
MLResult result = MLWorldRays.StartQ; 


this.worldRaysCallback = worldRaysCallback; 


virtual public void Update( 


Vector3 position, Vector3 direction, Vector3 up) 


{ 
if (MLWorldRays.IsStarted) 
{ 
queryParams.Position = position; 
queryParams.Direction = direction; 
queryParams.UpVector = up; 
MLWorldRays.GetWorldRays ( 
queryParams, HandleOnReceiveRaycast) ; 
$ 
$ 


virtual public void Stop() 


rf 
if (MLWorldRays.IsStarted) 
{ 
MLWorldRays.StopQ; 
} 
} 


virtual protected void HandleOnReceiveRaycast ( 
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MLWorldRays.MLWorldRaycastResultState state, 
Vector3 point, Vector3 normal, float confidence) 
{ 
bool hasRequestFailed = ( 
state == MLWorldRays.MLWorldRaycastResultState.RequestFailed) ; 
bool noCollision = ( 
state == MLWorldRays.MLWorldRaycastResultState.NoCollision) ; 


if C!hasRequestFailed && !noCollision) 


{ 
RaycastHit result = new RaycastHitQ); 
result.point = point; 
result.normal = normal; 
result.distance = Vector3.Distance( 
queryParams.Position, point); 
worldRaysCallback(state, result, confidence) ; 
$ 
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